ˆ The exam is closed book, closed calculator, and closed notes except your one-page crib sheet.

Transcription

1 CS Summer Introduction to Artificial Intelligence Midterm ˆ You have approximately minutes. ˆ The exam is closed book, closed calculator, and closed notes except your one-page crib sheet. ˆ Mark your answers ON THE EXAM ITSELF. If you are not sure of your answer you may wish to provide a brief explanation. All short answer sections can be successfully answered in a few sentences AT MOST. First name Last name SID edx username Name of person on your left Name of person on your right For staff use only: Q. Search / Q. Game Trees and Pruning / Q. Encrypted Knowledge Base /6 Q. MDPs - Farmland / Q. Reinforcement Learning / Total /

2 THIS PAGE IS INTENTIONALLY LEFT BLANK

3 Q. [ pts] Search Each True/False question is worth points. Leaving a question blank is worth points. Answering incorrectly is worth points. (a) Consider a graph search problem where for every action, the cost is at least ɛ, with ɛ >. Assume the used heuristic is consistent. (i) [ pt] [true or false] Depth-first graph search is guaranteed to return an optimal solution. False. Depth first search has no guarantees of optimality. Further, it measures paths in length and not cost. (ii) [ pt] [true or false] Breadth-first graph search is guaranteed to return an optimal solution. False. Breadth first search has no guarantees of optimality unless the actions all have the same cost, which is not the case here. (iii) [ pt] [true or false] Uniform-cost graph search is guaranteed to return an optimal solution. True. UCS expands paths in order of least total cost so that the optimal solution is found. (iv) [ pt] [true or false] Greedy graph search is guaranteed to return an optimal solution. False. Greedy search makes no guarantees of optimality. It relies solely on the heuristic and not the true cost. (v) [ pt] [true or false] A* graph search is guaranteed to return an optimal solution. True, since the heuristic is consistent in this case. (vi) [ pt] [true or false] A* graph search is guaranteed to expand no more nodes than depth-first graph search. False. Depth-first graph search could, for example, go directly to a sub-optimal solution. (vii) [ pt] [true or false] A* graph search is guaranteed to expand no more nodes than uniform-cost graph search. True. The heuristic could help to guide the search and reduce the number of nodes expanded. In the extreme case where the heuristic function returns zero for every state, A* and UCS will expand the same number of nodes. In any case, A* with a consistent heuristic will never expand more nodes than UCS. (b) Let h (s) be an admissible A* heuristic. Let h (s) = h (s). Then: (i) [ pt] [true or false] The solution found by A* tree search with h is guaranteed to be an optimal solution. False. h is not guaranteed to be admissible since only one side of the admissibility inequality is doubled. (ii) [ pt] [true or false] The solution found by A* tree search with h is guaranteed to have a cost at most twice as much as the optimal path. True. In A* tree search we always have that as long as the optimal path to the goal has not been found, a prefix of this optimal path has to be on the fringe. Hence, if a non-optimal solution is found, then at time of popping the non-optimal path from the fringe, a path that is a prefix of the optimal path to the goal is sitting on the fringe. The cost ḡ of a non-optimal solution when popped is its f-cost. The prefix of the optimal path to the goal has an f-cost of g + h = g + h (g + h ) C, with C the optimal cost to the goal. Hence we have that ḡ C and the found path is at most twice as long as the optimal path. (iii) [ pt] [true or false] The solution found by A* graph search with h is guaranteed to be an optimal solution. False. h is not guaranteed to be admissible and graph search further requires consistency for optimality. (c) [ pts] The heuristic values for the graph below are not correct. For which single state (S, A, B, C, D, or G) could you change the heuristic value to make everything admissible and consistent? What range of values are possible to make this correction? State: B Range: [,]

4 C h= S h=6 A h= B h= G h= D h=

5 Q. [ pts] Game Trees and Pruning You and one of the robots are playing a game where you both have your own score. ˆ The maximum possible score for either player is. ˆ You are trying to maximize your score, and you do not care what score the robot gets. ˆ The robot is trying to minimize the absolute difference between the two scores. In the case of a tie, the robot prefers a lower score. For example, the robot prefers (,) to (6,); it prefers (,) to (,); and it prefers (,) to (,). The figure below shows the game tree of your max node followed by the robots nodes for your four different actions. The scores are shown at the leaf nodes with your score always on top and the robots score on the bottom. (a) [ pts] Fill in the dashed rectangles with the pair of scores preferred by each node of the game tree. 6 (b) [ pts] You can save computation time by using pruning in your game tree search. On the game tree above, put an X on line of branches that do not need to be explored. Assume that branches are explored from left to right. (c) [ pts] You now have access to an oracle that tells you the order of branches to explore that maximizes pruning. On the copy of the game tree below, put an X on line of branches that do not need to be explored given this new information from the oracle. 6

6 Q. [6 pts] Encrypted Knowledge Base We have a propositional logic knowledge base, but unfortunately, it is encrypted. The only information we have is that: ˆ Each of the following boxes contains a propositional logic symbol (A, B, C, D, or E) or a propositional logic operator and ˆ Each line is a valid propositional logic sentence. X X X X X X 6 X X X X X X (a) [ pts] We are going to implement a constraint satisfaction problem solver to find a valid assignment to each box from the domain {A, B, C, D, E,,,,, }. Propositional logic syntax imposes constraints on what can go in each box. What values are in the domain of boxes -6 after enforcing the unary syntax constraints? Box Remaining Values A B C D E A B C D E A B C D E 6 A B C D E 6

7 (b) [ pts] You are given the following assignment as a solution to the knowledge base CSP on the previous page: A B A D C B D E Now that the encryption CSP is solved, we have an entirely new CSP to work on: finding a model. In this new CSP the variables are the symbols {A, B, C, D, E} and each variable could be assigned to true or false. We are going to run CSP backtracking search with forward checking to find a propositional logic model M that makes all of the sentences in this knowledge base true. After choosing to assign C to false, what values are removed by running forward checking? On the table of remaining values below, cross off the values that were removed. Symbol A B C D Remaining Values T F T F T F T E T F Forward checking removes the value false from the domain of B. Forward checking does not continue on to make any other arcs consistent. (c) [ pts] We eventually arrive at the model M = {A = F alse, B = F alse, C = T rue, D = T rue, E = T rue} that causes all of the knowledge base sentences to be true. We have a query sentence α specific as (A C) E. Our model M also causes α to be true. Can we say that the knowledge base entails α? Explain briefly (in one sentence) why or why not. No, the knowledge base does not entail α. There are other models for which the knowledge base could be true and the query be false. Specifically {A = F alse, B = F alse, C = T rue, D = T rue, E = F alse} satisfies the knowledge base but causes the query α to be false.

8 Q. [ pts] MDPs - Farmland In the game FARMLAND, players alternate taking turns drawing a card from one of two piles, PIG and COW. PIG cards have equal probability of being points or point, and COW cards are always worth points. Players are trying to be the first to reach points or more. We are designing an agent to play FARMLAND. We will use a modified MDP to come up with a policy to play this game. States will be represented as tuples (x, y) where x is our score and y is the opponent s score. The value V (x, y) is an estimate of the probability that we will win at the state (x, y) when it is our turn to play and both players are playing optimally. Unless otherwise stated, assume both players play optimally. First, suppose we work out by hand V, the table of actual probabilities. Opponent.... You... According to this table, V (, ) =., so with both players playing optimally, the probability that we will win if our score is, the opponent s score is, and it is our turn to play is.. (a) [ pts] At the beginning of the game, would you choose to go first or second? Justify your answer using the table. You should choose to go first. Since V (, ) =., if it is your turn and the scores are both, the probability that you will win is.. (b) [ pts] If our current state is (x, y) (so our score is x and the opponent s score is y) but it is the opponent s turn to play, what is the probability that we will win if both players play optimally in terms of V? V (y, x) (c) [ pts] As FARMLAND is a very simple game, you quickly grow tired of playing it. You decide to buy the FARMLAND expansion, BOVINE BONANZA, which adds loads of exciting cards to the COW pile! Of course, this changes the transition function for our MDP, so the table V above is no longer correct. We need to come up with an update equation that will ultimately make V converge on the actual probabilities that we will win. You are given the transition function T ((x, y), a, (x, y)) and the reward function R((x, y), a, (x, y)). The transition function T ((x, y), a, (x, y)) is the probability of transitioning from state (x, y) to state (x, y) when action a is taken (i.e. the probability that the card drawn gives x x points). Since we are only trying to find the probability of winning and we don t care about the margin of victory, the reward function R((x, y), a, (x, y)) is whenever (x, y) is a winning state and everywhere else. As in normal value iteration, all values will be initialized to (i.e. V (x, y) = for all states (x, y)). Write an update equation for V k+ (x, y) in terms of T, R and V k. Hint: you will need to use your answer from part b. V k+ (x, y) max T ((x, y), a, (x, y))[r((x, y), a, (x, y)) + ( V k (y, x ))] a x

9 Q. [ pts] Reinforcement Learning (a) Each True/False question is worth points. Leaving a question blank is worth points. Answering incorrectly is worth points. (i) [ pt] [true or false] Temporal difference learning is an online learning method. Temporal difference learning is used when we don t have the full MDP model and must collect online samples. (ii) [ pt] [true or false] Q-learning: Using an optimal exploration function leads to no regret while learning the optimal policy. In order to learn the optimal policy, you must explore, and exploring in general has a non-zero chance of regret. (iii) [ pt] [true or false] In a deterministic MDP (i.e. one in which each state / action leads to a single deterministic next state), the Q-learning update with a learning rate of α = will correctly learn the optimal q-values (assume that all state/action pairs are visited sufficiently often). Remember that the learning rate is only there because we are trying to approximate a summation with a single sample. In a deterministic MDP where s is the single state that always follows when we take action a in state s, we have Q(s, a) = R(s, a, s ) + max a Q(s, a ), which is exactly the update we make. (iv) [ pt] [true or false] A small discount (close to ) encourages greedy behavior. A discount close to zero will place extremely small values on rewards more than one step away, leading to greedy behavior that looks for immediate rewards. (v) [ pt] [true or false] A large, negative living reward ( ) encourages greedy behavior. A negative living reward adds a penalty for every step taken. If that penalty is large, the agent will prefer to find an exit as soon as possible despite potential rewards on longer paths. (vi) [ pt] [true or false] A negative living reward can always be expressed using a discount <. While both negative living rewards and discounts can encourage similar behavior, they are mathematically different. A discount has a multiplicative effect at each step, whereas a living reward only has an additive effect. (vii) [ pt] [true or false] A discount < can always be expressed as a negative living reward. While both negative living rewards and discounts can encourage similar behavior, they are mathematically different. A discount has a multiplicative effect at each step, whereas a living reward only has an additive effect. (b) [ pts] Given the following table of Q-values for the state A and the set of actions {F orward, Reverse, Stop}, what is the probability that we will take each action on our next move when we following an ɛ-greedy exploration policy (assuming any random movements are chosen uniformly from all actions)? Q(A, F orward) =. Q(A, Reverse) =. Q(A, Stop) =. Action Probability (in terms of ɛ) F orward Reverse Stop ( ɛ) + ɛ = ɛ ɛ ɛ